spiderpool-agent: support to configure the sysctl config for node #3772

cyclinder · 2024-07-25T10:55:03Z

Thanks for contributing!

What type of PR is this?

release/feature

What this PR does / why we need it:

support to configure the sysctl config for node

Which issue(s) this PR fixes:

Fixes #3587

Special notes for your reviewer:

{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:447","msg":"Setup sysctl","sysctl":"net.ipv4.neigh.default.gc_thresh3","value":"8192"}
{"level":"WARN","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:458","msg":"skip to setup sysctl","sysctl":"net.ipv4.neigh.default.gc_thresh3","value":"8192","error":"stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: no such file or directory"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:447","msg":"Setup sysctl","sysctl":"net.ipv6.neigh.default.gc_thresh3","value":"8192"}
{"level":"WARN","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:458","msg":"skip to setup sysctl","sysctl":"net.ipv6.neigh.default.gc_thresh3","value":"8192","error":"stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: no such file or directory"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:447","msg":"Setup sysctl","sysctl":"net.ipv4.conf.all.arp_notify","value":"1"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:450","msg":"success to setup sysctl","sysctl":"net.ipv4.conf.all.arp_notify","value":"1"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:447","msg":"Setup sysctl","sysctl":"net.ipv4.conf.all.forwarding","value":"1"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:450","msg":"success to setup sysctl","sysctl":"net.ipv4.conf.all.forwarding","value":"1"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:447","msg":"Setup sysctl","sysctl":"net.ipv6.conf.all.forwarding","value":"1"}
{"level":"DEBUG","ts":"2024-07-28T03:50:07.197Z","logger":"spiderpool-agent","caller":"cmd/daemon.go:450","msg":"success to setup sysctl","sysctl":"net.ipv6.conf.all.forwarding","value":"1"}

codecov · 2024-07-25T11:04:13Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.16%. Comparing base (cfae10b) to head (8e64eee).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3772      +/-   ##
==========================================
- Coverage   81.21%   81.16%   -0.05%     
==========================================
  Files          50       50              
  Lines        4391     4391              
==========================================
- Hits         3566     3564       -2     
- Misses        669      670       +1     
- Partials      156      157       +1

Flag	Coverage Δ
unittests	`81.16% <ø> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

charts/spiderpool/values.yaml

pkg/networking/sysctl/sysctl.go

weizhoublue · 2024-07-26T08:33:29Z

charts/spiderpool/values.yaml

@@ -454,6 +454,9 @@ spiderpoolAgent:
  securityContext: {}
  # runAsUser: 0

+  ## @param spiderpoolAgent.sysctlConfigs the sysctl configs of spiderpoolAgent pod


我在想，我们是否不提供该接口，spiderpool 不负责 sysctl 的其他职责，只是确保自身正常工作
只需要提供一个 disable tune sysctl 的参数即可，避免我们的默认值不符合预期。
其它的需求，用户自己去设置主机的 sysctl 写配置文件

spiderpool 不承担其它 sysctl 值的调优职责，也避免用户通过该接口设置了一些 sysctl，把 spiderpool 的相关代码跑挂了，我们是退出好，还是忽略好，都挺麻烦

我们可以保留这个接口，但不推荐用户使用此方式来来改任何 sysctl, 我们只是用它来做为我们自己的后门，方便以后要是出问题的时候，我们可以通过它快速配一下，避免需要重新编写代码

weizhoublue · 2024-07-26T08:36:24Z

pkg/networking/sysctl/sysctl.go

+		Value: "8192",
+	},
+	{
+		Name:  "net.ipv6.neigh.default.gc_thresh3",


上一行 comment，什么内核或高内核才有该 ipv6 的值。内核不存在该值时，会自动忽略

没搜到内核版本支持？

net.ipv6.neigh.default.gc_thresh3 (since Linux 2.2)
但是什么版本被拿掉了，要看下

就是没找到，内核代码没明显看出来

pkg/networking/sysctl/sysctl.go

weizhoublue · 2024-07-26T08:38:42Z

pkg/networking/sysctl/sysctl.go

+// DefaultSysctlConfig is the default sysctl config for the node
+var DefaultSysctlConfig = []struct {
+	Name  string
+	Value string


再加一些字段 ?
requiredKernelVersion bool 一些低内核下，一些值是不存在的，不需要设置，忽略它们是否设置成功
setOnIpv4 : true Spiderpool 工作在单双栈等场景下，只设置自己关心的值
setOnipv6: true

或者看其他项目是如何实践的

kube-proxy 会判断kernel 版本，这个有点复杂了，calico 会通过是否启用 ipv6 来决定配置 ipv6 的参数

这里如果需要判断 kernel 版本和是ipv4 或 ipv6，就有点繁琐了，这里应该是尽量去设置，忽略可接受的失败，否则如 ipv6-only 下还要去判断 ipv4 的 sysctl 不能去设置

spidepool helm 安装选项中有是否启用 ipv4 还是 ipv6 的环境变量传递进来

cmd/spiderpool-agent/cmd/daemon.go

weizhoublue · 2024-07-31T06:23:51Z

charts/spiderpool/README.md

-| `ipam.spidersubnet.enable`                            | SpiderSubnet feature.                                                                            | `true` |
-| `ipam.spidersubnet.autoPool.enable`                   | SpiderSubnet Auto IPPool feature.                                                                | `true` |
-| `ipam.spidersubnet.autoPool.defaultRedundantIPNumber` | the default redundant IP number of SpiderSubnet feature auto-created IPPools                     | `1`    |
+| `ipam.spiderSubnet.enable`                            | SpiderSubnet feature.                                                                            | `true` |


虽然是带货一些小错误，但要动这些字段，注意 doc 下一些文档要纠正

weizhoublue · 2024-07-31T06:24:30Z

charts/spiderpool/README.md

@@ -124,6 +124,7 @@ helm install spiderpool spiderpool/spiderpool --wait --namespace kube-system \
 | `global.ipamUNIXSocketHostPath` | the host path of unix domain socket for ipam plugin                         | `/var/run/spidernet/spiderpool.sock` |
 | `global.configName`             | the configmap name                                                          | `spiderpool-conf`                    |
 | `global.ciliumConfigMap`        | the cilium's configMap, default is kube-system/cilium-config                | `kube-system/cilium-config`          |
+| `global.tuneSysctlConfig`       | enable set specify sysctl configs for each node.                            | `false`                              |


这个就是 spdieragent.tuneSysctlConfig 毕竟合适，它不生效到其它组件

weizhoublue · 2024-07-31T06:25:29Z

charts/spiderpool/templates/daemonset.yaml

@@ -245,15 +245,11 @@ spec:
        {{- with .Values.spiderpoolAgent.extraEnv }}
        {{- toYaml . | nindent 8 }}
        {{- end }}
-        {{- if or .Values.dra.enabled .Values.spiderpoolAgent.securityContext }}
        securityContext:


这个默认是强制的？难免有一些安全风险和背锅

哦，这个应该要恢复

weizhoublue · 2024-07-31T06:26:50Z

docs/reference/spiderpool-agent.md

+
+| sysctl config | value | description |
+| -------------| ------| ------------|
+| net.ipv4.neigh.default.gc_thresh3 | 8196 | This is the hard maximum number of entries to keep in the ARP cache. The garbage collector will always run if there are more than this number of entries in the cache. for ipv4  |


8196 -> 20480
省得又不够

文档需要更新

weizhoublue · 2024-08-01T02:56:31Z

charts/spiderpool/values.yaml

@@ -450,6 +450,9 @@ spiderpoolAgent:
      ## @param spiderpoolAgent.resources.requests.memory the memory requests of spiderpoolAgent pod
      memory: 128Mi

+  ## @param spiderpoolAgent.tuneSysctlConfig enable set specify sysctl configs for each node.
+  tuneSysctlConfig: false


这个默认是 true 把？

#3772 (comment)

我怕打开会覆盖客户的默认值？

? tuneSysctlConfig: true

这些事 underlay 必要条件，应该是默认的

weizhoublue · 2024-08-01T02:58:43Z

charts/spiderpool/README.md

@@ -282,6 +282,7 @@ helm install spiderpool spiderpool/spiderpool --wait --namespace kube-system \
 | `spiderpoolAgent.resources.limits.memory`                                            | the memory limit of spiderpoolAgent pod                                                          | `1024Mi`                                   |
 | `spiderpoolAgent.resources.requests.cpu`                                             | the cpu requests of spiderpoolAgent pod                                                          | `100m`                                     |
 | `spiderpoolAgent.resources.requests.memory`                                          | the memory requests of spiderpoolAgent pod                                                       | `128Mi`                                    |
+| `spiderpoolAgent.tuneSysctlConfig`                                                   | enable set specify sysctl configs for each node.                                                 | `false`                                    |


enable to set required sysctl on each node to run spiderpool. refer to http:/.....doc... for details

weizhoublue · 2024-08-01T03:01:01Z

cmd/spiderpool-agent/cmd/daemon.go

@@ -75,6 +76,15 @@ func DaemonMain() {
 	}
 	logger.Sugar().Infof("Spiderpool-agent config: %+v", agentContext.Cfg)

+	// setup sysctls
+	if agentContext.Cfg.TuneSysctlConfig {
+		if err := sysctlConfig(agentContext.Cfg.EnableIPv4, agentContext.Cfg.EnableIPv6); err != nil {


作为 info 级别的日志，打开时，看不到一行 " set syctl " 日志

这个日志是有的

https://github.com/spidernet-io/spiderpool/pull/3772/files#diff-901dd9b896967a0756a749c0c5afa309570d80f088f5ee34c144c18f0b2ca80bR450 改为info

weizhoublue · 2024-08-01T03:02:03Z

pkg/networking/sysctl/sysctl.go

+	if _, err := sysctl.Sysctl(sysConfig, value); err != nil {
+		return err
+	}
+


需要一行 info 级别日志： sysctl 什么 key 设置了什么值

https://github.com/spidernet-io/spiderpool/pull/3772/files#diff-901dd9b896967a0756a749c0c5afa309570d80f088f5ee34c144c18f0b2ca80bR450 改为 info 即可

Signed-off-by: cyclinder <qifeng.guo@daocloud.io>

spiderpool-agent: support to configure the sysctl config for node Signed-off-by: robot <tao.yang@daocloud.io>

…pt_sysctl spiderpool-agent: support to configure the sysctl config for node

cyclinder added cherrypick-release-v0.9 cherrypick-release-v1.0 Cherry-pick the PR to branch release-v1.0. labels Jul 25, 2024

cyclinder requested a review from weizhoublue as a code owner July 25, 2024 10:55

cyclinder force-pushed the spiderpoolagent/opt_sysctl branch from 7b169b1 to ba63403 Compare July 25, 2024 10:58

cyclinder force-pushed the spiderpoolagent/opt_sysctl branch from ba63403 to 264826f Compare July 25, 2024 12:56